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INTRODOCTION 


The  proliferation  of  the  scientific  literature  in  recent  years 
has  encouraged  the  application  of  computer  technology  to  such 
traditional  human  endeavors  as  information  analysis  and  problem 
solving.  Within  the  biological  sciences,  for  example,  the  knowledge 
base  reportedly  doubles  every  five  years  (Ziman,  1980).  As  a  result, 
domain  experts  have  become  increasingly  inundated  with  new 
information . 

Computer  technology,  particularly  the  field  of  artificial 
intelligence,  has  been  shown  to  hold  considerable  promise  for  dealing 
with  this  problem.  This  has  been  especially  true  in  the  realm  of 
medicine  where  literature  proliferation  has  become  an  immense  problem 
for  the  line  physician.  Consequently,  it  was  appropriate  that  one  of 
the  first  successful  knowledge  based  expert  systems,  MYCIN,  was 
developed  for  specialized  medical  diagnosis  and  treatment  (Shortliffe, 
1976)  . 

Prerequisite  for  effective  use  of  intelligent  machines  such  as 
MYCIN  is  the  design  of  user/machine  interfaces  that  encourage  user 
acceptance  and  that  optimize  user/machine  performance.  Accordingly, 
this  paper  describes  empirical  research  investigating  the  cognitive 
psychology  of  user  interactions  with  the  MYCIN  system.  This  research 
was  performed  in  the  context  of  a  more  general  research  program  that 
focused  on  advancing  a  general  understanding  of  the  psychology  of  user 
interactions  with  intelligent  machines  and  that  also  endeavored  to 
generate  design  principles  that  could  lead  to  optimal  user  engineering 
of  future  expert  systems. 


The  central  theory  of  the  current  study  is  that  a  user's  ability 
to  solve  problems  in  conjunction  with  an  intelligent  system  depends 
upon  the  user's  mental  model  of  that  system's  operating  processes. 
Three  experiments  reported  in  a  previous  study  (Lehner  &  Zirk,  1985) , 
strongly  supported  this  idea.  In  the  Lehner  and  Zirk  experiments, 
problem  solving  with  a  generic  expert  system,  ERS ,  Embedded  Rule  Based 
System  (Barth,  1984)  using  a  stock  market  game  domain  was  shown  to  be 
significantly  enhanced  by  an  accurate  mental  model  of  system 
operation . 

Although  this  mental  model  concept  appeared  to  be  supported  by 
the  previous  experiments,  it  acquired  support  within  the  confines  of 
an  artificial  domain  whereas  the  present  study  examined  the  impact  of 
mental  model  on  problem  solving  performance  within  the  domain  of  real 
world  medical  consultations. 

In  the  case  of  expert  systems  like  MYCIN,  mental  model  can  be 
understood  in  terms  of  two  general  categories:  machine  understanding 
and  content  understanding.  For  instance,  machine  understanding  refers 
to  recognition  that  (1)  knowledge  is  encoded  in  rules,  (2)  rules  tend 
to  be  organized  as  inference  networks,  and  (3)  that  the  problem 
solving  processes  are  goal  driven,  (i.e.,  MYCIN  selects  a  potential 
diagnosis  and  then  tests  symptomology  against  medical  laboratory  test 
results  to  confirm  or  disprove  it.)  Content  understanding  refers  to 
the  extent  to  which  the  user  knows  the  specific  heuristics  that  have 
been  encoded  into  an  expert  system.  In  the  present  study,  the  impact 


of  mental  model  on  the  problem  solving  performance  of  domain  experts 
was  investigated  through  the  presentation  of  mental  model  -  specific 
instructional  material  in  terms  of  machine  understanding. 


METHOD 


Subjects 

Six  second  and  third  year  medical  students  from  Howard  Medical 
School  and  the  George  Washington  University  Medical  School 
participated  as  paid  participants  in  the  current  study.  The  mean  age 
was  26  with  a  range  of  24  to  30  years.  None  of  the  participants  had 
prior  experience  with  rule-based  systems  or  computer  aided  problem 
solving  tasks. 

Materials 

A  rule-based  expert  system,  MYCIN  (Shortliffe,  1974),  was  used  as 
the  intelligent  interface  testbed  for  the  current  study.  MYCIN  is  a 
backward  chaining  heuristic  medical  consultation  system  that  provides 
advice  to  physicians  on  the  diagnosis  of  bacterial  infections  and 
appropriate  drug  therapy.  The  data  base  in  this  case  is  the  physician 
himself  providing  information  in  response  to  questions  generated  by 
the  program.  The  Mycin  program  contains  several  hundred  rules  with 
attached  certainty  factors  of  the  form  "if  premise  then  action"  within 
which,  antibiotic  treatment  knowledge  is  encoded  and  against  which  the 
patient's  state  and  initial  laboratory  findings  are  tested.  Rules 
with  matching  conditions  are  subsequently  executed  and  the  final 
result  is  a  set  of  rank  ordered  therapy  recommendations. 

The  MYCIN  system  consists  of  three  interrelated  programs.  An 
overall  diagram  of  MYCIN  and  the  relationship  between  the  component 
programs  is  shown  in  Figure  1. 
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Figure  1 


The  Consultation  program  serves  as  the  core  of  the  system;  it 
interacts  with  the  physician  to  obtain  information  about  the  patient, 
generating  diagnoses  and  therapy  recommendations.  The  Explanation 
program  provides  explanations  and  justifications  for  the  program's 
actions  while  the  Knowledge-Acquisition  program  allows  domain  experts 
to  update  the  system's  knowledge  base. 

MYCIN  is  written  in  Interlisp-D  and  currently  runs  on  Digital 
Equipment  Corporation  DEC  System-20  computer  systems  using  either  the 
VMS  or  UNIX  operating  systems. 

For  the  present  study,  an  APPLE  HE  computer  was  remotely  linked 
to  MYCIN  at  Stanford  Medical  Center's  SUMEX-AIM  computer  via  TYMNET. 
Experimental  Design 

The  two  levels  of  the  independent  variable  were  (1)  mental  model 
and  (2)  no  mental  model.  The  dependant  measure  was  task  performance 
as  measured  by  the  number  of  system  queries  used  for  each  of  the  four 
problems.  Subjects  in  the  mental  model  condition  received  as  part  of 
their  instructions  a  written  and  illustrated  description  of  rules, 
inference  networks  and  backward  chaining  procedures.  This  section 
described  the  structure  of  a  general  inference  network,  explained  how 
the  MYCIN  system  identified  goals,  intermediate  hypotheses,  and  data 
items,  and  chained  up  and  down  the  network  to  obtain  degree  of 
certainty  values  for  each  goal.  Included  in  this  section  was  a 
pictorial  display  of  an  inference  net  and  a  simple  example  of  its 
operation.  By  working  through  this  section,  the  user  presumably 


developed  a  mental  model  of  how  the  expert  system  solved  problems. 
Subjects  in  the  no  mental  model  condition  were  only  afforded  a  cursory 
explanation  concerning  machine  knowledge. 


Test  Bed  Domain 


The  experimental  domain  was  a  medical  consultation  system,  MYCIN. 
Task  problems  were  constructed  by  creating  four  completed  consultation 
sessions  for  which  a  number  of  heuristics  were  employed  by  MYCIN  to 
generate  diagnostic  heirarchies  and  their  accompanying  treatment 
recommendations  (a  sample  MYCIN  consultation  is  contained  in  appendix 
A).  Thus,  for  each  of  the  four  problem's  set  of  rules,  there  was  only 
one  rule  responsible  for  the  primary  diagnosis  and  it  was  that  rule 
the  subject  was  instructed  to  find. 

Procedure 

Subsequent  to  reading  a  description  of  the  experiment,  subjects 
received  the  instruction  booklet  pretaining  to  their  particular  mental 
model  condition  assignment.  These  instructions  specified  the 
objectives,  procedures,  and  requirements  of  the  problem  solving  task. 
Subjects  were  seated  at  a  table  directly  in  front  of  the  terminal  with 
ample  space  to  arrange  their  individual  work  sheets.  Upon  completion 
of  the  experimental  booklet,  the  subjects  were  afforded  a  walk-through 
practice  consultation  with  MYCIN  and  shown  how  to  access  the  rules 
which  had  been  used  to  generate  the  system's  theraputic 

recommendations . 

Performance  Measures 

For  each  of  four  consultation  problems,  subjects  were  given  a 
printout  of  a  previous  consultative  session  and  a  sequential  list  of 
the  rules  MYCIN  used  to  reach  its  diagnostic  and  theraputic 

conclusions.  The  subjects  were  then  instructed  to  interact  with  the 


MYCIN  system  in  order  to  examine  those  rules  in  any  order  that  they 
saw  fit  until  they  had  isolated  the  particular  rule  responsible  for 


each  of  the  four  predetermined  primary  diagnoses.  The  number  of 
"rule"  commands  was  recorded  for  each  of  the  four  problems  for  each 
subject . 

A  second  performance  measure  was  the  ten  item  subjective 
questionaire .  Subjects  indicated  on  a  ten  point  scale  from  0  ("very 
strongly  disagree")  to  10  ("very  strongly  agree")  their  agreement  with 
statements  addressing  (1)  the  understanding  of  MYCIN's  operating 
procedures,  (2)  the  ease  of  system  use,  (3)  the  confidence  of  final 
user  decisions,  and  (4),  the  adequacy  of  the  time  allotment. 

A  second  questionaire  was  also  employed  which  was  designed  to 
determine  which  components  of  the  mental  model  (in  terms  of 
instruction  booklet  sections  which  compartmentalized  the  concepts  of 
rules,  inference  networks,  and  backward  chaining  procedures)  were  of 
most  value  to  the  participants  in  understanding  and  performing  their 
tasks . 

PRELIMINARY  RESULTS 

Due  to  difficulty  in  securing  the  participation  of  medical 
students  (whose  intensive  training  and  time  consuming  professional 
responsibilities  precluded  easy  accessabi 1 i ty ) ,  only  six  subjects  were 
actually  recruited  for  the  current  study.  However,  performance 
differences  were  observed  between  the  two  mental  model  conditions. 
The  beneficial  impact  of  an  accurate  mental  model  on  the  problem 
solving  tasks  in  terms  of  system  queries  is  exemplified  in  Figure  2  by 
the  mean  scores  of  the  two  experimental  groups. 
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In  addition  to  the  mean  differences,  a  T-test  revealed 
significant  differences  in  user  performance  between  the  two  mental 
model  conditions  in  terms  of  their  use  of  the  "rule"  command 
T(4)=1.94,  .l>p>.05,lt,sig. 

Further  data  analysis  was  performed  by  evaluating  subject's 
responses  to  the  10-item  questionai re .  Users  receiving  accurate 
mental  models  reported  greater  "understanding  of  the  system's 

operating  procedures",  than  did  those  in  the  no  mental  model  group 
with  means  of  7.3  and  5.4  respectively.  Reports  of  "ease  of  system 
use"  followed  the  predicted  pattern,  the  means  being  7.0,  and  6.5. 
There  was  no  difference  between  the  two  groups  in  terms  of  "confidence 
of  final  user  decisions"  and  "time  allotment  adequacy". 

Responses  to  the  second  questionaire  concerning  the  task 
relevance  of  the  instructional  materials  revealed  that  their  knowledge 
of  rules,  inference  networks,  and  backward  chaining  processes  had  been 
the  most  helpful  in  terms  of  subsequent  task  performance.  On  the 
other  hand,  the  no  mental  model  group  felt  that  the  sample 
consultation  had  been  the  most  helpful  to  them  in  terms  of  their  task 
performance . 


V, 


DISCOSSION 


In  general,  the  preliminary  results  of  this  study  seem  to  agree 


with  the  basic  theory  that  the  quality  of  user/expert  system  problem 
solving  is  driven  to  a  large  degree  by  the  user's  mental  model  of  the 
expert  system's  problem  solving  procedures.  Highest  problem  solving 


performance  was  achieved  by  subjects  having  a  good  mental  model,  i.e., 
those  having  an  understanding  of  the  system's  use  of  rules,  inference 
networks  and  backward  chaining  procedures.  The  basic  explanation  for 
this  is  that  a  user  with  an  accurate  mental  model  is  well  suited  to 
interact  with  an  expert  system  for  which  he  or  she  has  a  conceptual 
grasp. 

Future  research  efforts  should  continue  to  focus  on 
user/intelligent  machine  interaction  using  "real-world"  systems  such 
as  MYCIN.  The  information  gleaned  from  such  efforts  are  likely  to 
contribute  toward  the  development  of  a  theoretical  base  of  user-expert 
system  interaction  which  in  turn  would  permit  the  design  of  more 
optimal,  hence  user  acceptable,  user/ intell igent  machine  interfaces. 
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- PrtTl  ENT-321 - 

I >  Pat  I vn  t  «  n«m*  t 
•*  PT321 
2>  Ag*i 
••  3  YEARS 

3)  S«K I 
••  FEMALE 

4)  R<c»i 

••  CAUCASIAN 

3)  Ar«  th*r*  *o>'  cultures  tor  Pt321  which  may'  be  rslatad  to  ths 
prsstnt  illness,  and  tram  which  organisms  have  been  grown 
sueeesstully  in  the  microbiology  laboratory? 

ee  NO 

6>  Are  there  any  pending  cultures  tor  Pt321? 
ee  YES 


- CULTURE-1 - 

7)  From  what  site  was  the  specimen  tor  CULTURE-1  taken? 
ee  CSF 

8)  Please  give  the  date  and  time  when  the  pending  cat  culture 
< CULTURE-1 >  was  obtained.  <mo/da/yr  time) 

ee  l7-Mar-74  t2i02PM 

9)  Uas  a  smear  examined  at  the  time  that  the  specimen  tor  the 
pending  cst  culture  <CULTURE-I>  was  obtained? 

ee  res 

IOj  Uere  any  organisms  seen  on  the  smear  ot  the  specimen 
submitted  as  the  pending  cst  culture  (CULTURE-1 >? 
ee  YES 

- ORGANISM-1 - 

11)  The  stain  (Oram,  India  ink,  or  Z i ehl -Neel  sen  acid-tast) 
at  OROANISM-It 

ee  GRMdtEG 

12)  Is  ORGANlSM-1  a  rod  or  coccus  (etc.ll 
ee  ROO 

13)  uhat  IS  the  torm  ot  the  individual  organisms  (e.g. 
Lancet-shaped  tor  cocci,  tusitorm  tor  rods,  etc.)? 

ee  UNKNOUN 

14)  «ny  other  organisms  noted  on  the  smear? 
ee  NO 

13)  Are  there  any  other  pending  cultures? 
ee  NO 

10)  Have  there  been  negative  (as  opposed  to  pending)  cultures 
obtained  tram  Pt32l  which  may  be  relevant  to  the  present 
I  I  I  ness? 
e*  NO 

17)  Oo  you  suspect  Pt32l  may  have  an  intection  at  a  site  tram 
which  you  have  not  obtained  culture  specimens? 


if..'  ■  .1 

'A 


■f. . 


18)  Is  Pt32l  currently  receiving  therapy  with  any  antimicrobial 
agent’ 

•  e  NO 

19)  During  the  presen  t^  i  1 1  ness,  has  Pt321  been  treated  wi  th  an 
antimicrobial  agent  which  is  no  longer  being  atPni  n  i  stered? 

•  e  NO 

2U)  Has  Pt32l  recently  had  symptoms  ot  persistent  headache  or 
other  abnormal  neurologic  symptoms  (dizziness,  lethargy, 
etc  . )  ) 

•  •  YES 

21)  Has  Pt32l  recently  had  objective  evidence  ot  abnormal 

neurologic  signs  (nuchal  rigidity,  coma,  seizures,  etc.) 
documented  bv  ohvsician  obser..*tion  or  examination) 


A-2 


9^  f 


•  •  YES 


Th*  CSF  culture*  will  b*  coA*ider*d  to  b*  ***oci*t*d  Kuth  MENINGITIS. 
22)  PI****  giu*  th*  d*t*  on  which  clinical  *wid*nc*  t •xmp tom* , 

*ign*,  or  l*bor*tor>i  t**t*>  oi  th*  meningiti*  tirst  appeared. 
**  17-M*r-7P  «i02PM 

Although  all  o^  the  CSF  culture*  were  obtained  betor*  th*  clinical 
euidenc*  ot  MENINGITIS  began,  I  wi I  I  conaider  *or  therapr  tho** 
that  were  obtained  within  4B  hour*  o^  th*  onset. 

23>  1*  th*  meningiti*  a  hoap i ta  I -acqu i red  infection? 

a*  NO 

24)  Ha*  Pt321  had  any  injury  or  insult  to,  or  defect  In  the 
central -neruous-system  texcluding  neurosurgery)? 

a*  UNKNOUN 

25)  On  a  scale  of  0  to  4,  where  higher  number*  indicate  increasing 
severity,  how  would  you  rate  Pt321's  degree  of  •icKness? 

aa  4 

24)  Is  Pt321  febrile? 
as  YES 

27)  Do  you  know  the  results  of  a  recent  CBC  of  Pt321? 
aa  YES 

28)  Uhat  was  Pt321's  peripheral  whit*  count  (in  thousands)? 
a*  3.4 

29)  Percent  of  UBC't  which  were  PfN'si 
a*  7? 

30)  Percent  of  th*  peripheral  UBC's  which  are  imsatur*  formsi 
as  I  2 

31)  Is  Pt321  a  compromised  host  (e.g.  alcoholic,  sickle-cel  I- 
disease,  immunosuppressed)? 

a*  NO 

(Considering  organisms  (other  than  these  isolated  from  cultures) 
which  might  be  present...) 

32)  Does  Pt321  have  an  *bnorm»l  chest  X-ray? 
a*  NO 

33)  Does  Pt32l  have  on*  or  more  of  th*  following  risk  factors  for 
tbi  a)  positive  intermediate  strength  PPD,  b)  history  of  close 
contact  with  a  person  with  active  tb,  c>  a  household  member 
with  a  past  history  of  active  tb,  d>  apical  scarring  on  chest 
xray,  e>  history  of  granulomas  on  biopsy  of  liver,  lymph  node, 
or  other  organ*.’ 

a*  NO 

34)  For  how  many  day*  has  Pt32I  had  abnormal  neurologic  signs? 
a*  ?.4  HOURS 

33)  For  how  many  days  has  Pt32l  had  neurologic  symptoms? 
a*  12  HOURS 

34)  Ha*  Pt32l  had  recent  evidence  of  tender  or  enlarged  salivary 
gl and* ’ 

a*  NO 

37)  Has  Pt32l  been  exposed  to  any  contagious  disease  recently 
^e.g.  meningococcal  disease,  mumps)  ? 

as  UNKNOUN 

38)  Ooe*  Pt321  have  any  cutaneous  lesions  or  rash  on  physical 
exami nat ion? 

aa  NO 

3?)  Uhat  I*  th*  protein  value  <mg/IOOcc)  in  th*  CSF? 
a*  324 

40)  Uhat  1*  th*  total  UbC  count  of  th*  CSF? 
a*  1000 

41)  Uhat  percent  of  th*  UBC*  in  th*  CSF  were  polys? 
as  100 

42)  Uhat  1*  th*  glucose  value  (milligrams  percent)  in  th*  CSF’ 

•  a  113 

43)  Uhat  IS  th*  simultaneous  blood  glucose? 
a*  220 

44)  Doe*  Pt321  have  evidence  of  ocular  nerve  dysfunction? 

•  a  NO 


T 

•  N." 


I  t 


■  r-. 


!  V. 


.V 

A' 


‘-S 

l>  V 


Ik 


A 


43>  Has  Pt321  awsr  untfargons  any  typa  of  surgtry? 
aa  UNKNOUN 

44)  Ooas  Pt321  llwa  in  a  crowdad  anwironmant? 

aa  NO 

47)  ts  Pt321's  clinical  history  consistant  with  EPIOLOTTITIt? 
aa  UNKNOWN 

48)  Is  Pt321'’s  clinical  history  consistant  with  OTITtS-NEOIA? 
as  UNM40UN 

49)  ts  Pt321  a  burn  pat  I  ant? 
aa  UNKNOWN 

50)  Doas  Pt321  haws  4ocal  naurelooical  signs? 
aa  UNIOfOWN 

COatarsilning  which  drugs  ara  daslrabla  for  usa  against  tha 
Haaioph  1 1  us- 1  nf  I  uaniaa . . .  ] 

9t'0*t1^s  Pt32l  haws  a  clinically  significant  alltrgic  raactlon  to 
any riantlailcrobial  agant? 

so  NO 
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CConsidaring  ANPICILLtN-ANO-CHLORPNPH8flCOL  for  usa  against 
INFECTION-1...] 

52)  Oo  you  hava  raason  to  suspact  that  Pt32t  may  hava  impairad 
ranal  function? 

aa  NO 

53)  Pt321's  waight  in  kilograms  <or  tnumbar)  POUNDS) i 

as  20 

CREC-ll  My  prafarrad  tharapy  racommandation  is  as  followsi 
Oiva  tha  following  in  combination! 

I)  AMPICILLIN 

Dosai  t.Og  <8.0  ml)  qOh  IV  Icalculatad  on  basis  of  SO  rngi'kg] 
(Tha  dosa  should  ba  modiflad  in  SEVERE  ranal  failurt.  I  can 
adjust  tha  dosa  only  if  a  craatinina  claaranca  or  a 
sarum  craatinina  is  obtained. ] 
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2)  CHLORAMPHENICOL 

Dosa I  SOO  mg  <S.O  ml)  gOh  IV  Icalculatad  on  basis  of  2S 
mgi'kgl 
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Washington,  D.  C.  20301 

Dr.  Dennis  Eeedom 

Office  of  the  Deputy  Under  Secretary 
of  Defense  (C  1) 

Pentagon 

Washington,  D.  C.  20301 
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Office  of  Naval  Research 
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>> 

Office  of  Naval  Research 

1 

ii^ 

hr 

800  North  Quincy  Street 
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Naval  Research  Laboratory 
Washington,  D.C.  2037S 
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Operations  Research  Department 
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Dean  of  Research  Administration 
Naval  Postgraduate  School 
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Engineering  &  Computer  Science 
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Naval  Ocean  Systems  Center 
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Kaval  Ocean  Systems  Center 

Hawaii  Laboratory 
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800  N.  Quincy  Street 

Arlington,  VA  22217 


CDR  Ton  Jones 
Naval  Air  Systems  Coiranand 
Human  Factors  Programs 
NAVAIR  330J 

Washington,  D.  C.  20361 
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